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In the report the approach to estimation of quality of planned experiments is considered. This approach is based 
on the analysis of uncertainty, which will take place under the future hypotheses testing about the existence of 
a new phenomenon in Nature. The probability of making a correct decision in hypotheses testing is proposed 
as estimator of quality of planned experiments. This estimator allows to take into account systematics and 
statistical uncertainties in determination of signal and background rates. 



1. Introduction 

One of the common goals in the forthcoming ex- 
periments is the search for new phenomena. In es- 
timation of the discovery potential of the planned 
experiments the background cross section (for exam- 
ple, the Standard Model cross section) is calculated 
and, for the given integrated luminosity L, the av- 
erage number of background events is rib = cr& ■ L. 
Suppose the existence of new physics leads to addi- 
tional nonzero signal cross section a s with the same 
signature as for the background cross section that re- 
sults in the prediction of the additional average num- 
ber of signal events n s = a s ■ L for the integrated 
luminosity L. The total average number of the events 
is < n >= n s + rib = (os + o&) • L. So, as a result 
of new physics existence, we expect an excess of the 
average number of events. Let us suppose the proba- 
bility of the realization of n events in the experiment 
is described by function /(n; A) with parameter A. 

In the report the approach to estimation of quality 
of planned experiments is considered. This approach 
is based on the analysis of uncertainty, which will take 
place under the future hypotheses testing about the 
existence of a new phenomenon in Nature. 

We consider a statistical hypothesis 

Hq: new physics is present in Nature 

against an alternative hypothesis 

Hi : new physics is absent in Nature. 

The value of uncertainty is defined by the values of 
the probability to reject the hypothesis Hq when it is 
true (Type I error) 

a = P(reject Hq\Hq is true) 

and the probability to accept the hypothesis Hq when 
the hypothesis Hi is true (Type II error) 

(3 = P (accept Hq\Hq is false). 

Here a is a significance of the test and 1 — [3 is a power 
of the test. 



We propose to use as estimator of the quality of 
planned experiments the probability of making a cor- 
rect decision in the future hypotheses testing 1 — k 



1 - k = 1 



a + (3 



(1) 



and as estimator of the distinguishability of the hy- 
potheses 1 — K 



a + (3 



2 - (a + 0) ' 



(2) 



where d and (3 are the estimators of Type I error a 
and Type II error (3 calculated by the applying of the 
equal-tailed test (a = (3). 

The 1 — k is the estimator of quality of planned ex- 
periments. This estimator allows to take into account 
systematics and statistical uncertainties [2j in deter- 
mination of signal and background rates. The 1 — k 
have no dependence on the choice what is Hq, what is 
H±. This value is free from restrictions of such type. 
It is an advantage of our approach. We also propose 
to use an equal probability test Q as a good approx- 
imation of the equal-tailed test for estimation of the 
probabilities a and (3 in the case of Poisson distribu- 
tions. 



2. What is meant by the probability of 
making a correct decision in hypotheses 
testing? 

Suppose that the probability of the realization of 
n events in experiment is described by the function 
/(n; A) with parameter A and we know the expected 
number of signal events n s and expected number of 
background events n&. 

Let us determine what we mean by the probabil- 
ity of making a correct decision under the future hy- 
potheses testing about the presence or absence of the 
new phenomenon in Nature in case of carrying out 
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Figure 1: Equal probability test for the case n;, = 11 and 
n a — 10.61 gives the critical value n c = 16 and, 
correspondingly, the probability of uncorrect decision 
k = 0.09 and the measure of distinquishability of 
hypotheses k — 0.1. 



the planned experiment. Let us use the frequentist 
approach, i.e. consider all the possible results of the 
experiment in cases when both the hypothesis Hq is 
true or the hypothesis H± is true, define the criterion 
for the hypothesis choice and calculate the probability 
of making a correct decision. It is possible, because 
we construct the critical area in such a way that the 
probability of incorrect and, correspondingly, correct 
choice in favour of one of the hypothesis have no de- 
pendence on whether true is Hq or Hi . So, we will con- 
sider 2 conditional distributions of probabilities (see, 
Fig.l) 



/o(n) 
h(n) 



f(n; n s + n b ) 
f(n;n b ) 



(3) 



making numerical calculations. We suppose that any 
prior suppositions about Hq and Hi can be included 
to / (n) and /i(n). 

After choosing a critical value (or a critical area) 
same way, it is possible to count up the estimators of 
Type I error (a) and Type II error (f3). 

In the case of applying the equal-tailed test their 
combination 







(4) 



is the probability of making incorrect choice in favour 
of one of the hypothesis. 

The explanation is very simple. In actuality we 
must estimate the random value K = a + [3 = k + e, 
where k is a constant term and e is a stochastic term. 



The a is a fraction of incorrect decisions if the hy- 
pothesis Hq takes place. In this case the is ab- 
sent because the hypothesis Hi is not realised in Na- 
ture. Correspondingly, the is a fraction of incorrect 
decisions if the hypothesis Hi takes place. In this 
case the a is absent. Let us the hypothesis Hq be 
true then the Type I error equals a and the error of 

/Tn . „ „ „ a + (3 
our estimator (Lq.4) is e = 



a 







If hypothesis Hi is true then the Type 
II error equals (3 and the error of the estimator is 



= 







a- 



stochastic term takes the values ±- 



By this mean the 

t-0 



and if we re- 



quire a = then both errors of the estimation are 
equal to (k — a = k — (3 = 0). As a result the 
estimator (Eq.4) gives the probability of making an 
incorrect decision in future hypotheses testing. Re- 
ally, if a = and the Hq takes place in Nature then 
k = (a + 0)/2 — 2 • a/2 = a. In the same manner 
k = if the Hi takes place. Accordingly, 1 — k is the 
probability to make a correct choice under the given 
critical value. 

Under the hypotheses testing we can also estimate 
the measure 1 — k of distinguishability of the hypothe- 
ses Hq and Hi 1 by the calculation of 



a + 
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2 - (a + 0) 
There are 3 possibilities. 

• Distributions fo(n) and /i(n) have no overlap- 
ping, hence, the distributions are completely dis- 
tinguishable and any result of the experiment 
will give the correct choice between hypotheses, 
i.e. k = 0. 

• Distributions fo(n) and fi(n) coincide com- 
pletely. It means, that it is impossible to get 
a correct answer, i.e. fo(n) and fi{n) are not 
distinguishable, i.e. k = 1. 

• Distributions fo(n) and fi(n) do not coincide, 
but they have an overlapping, i.e. k is ratio 



x If we will use the geometric approach (let us the A is a set 
of possible realizations of the result of the planned experiment 
if the hypothesis Ho takes place in Nature and the B is a set of 
possible realizations of the result of the planned experiment if 
the hypothesis Hi takes place) then we have the total number 
of the possibilities for decision equals to A B and the fraction 

Af]B a + p 



of incorrect decisions will be 



A{]B 



(& + $) 
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of the probability of making incorrect choice to 
probability making correct choice in favour of 
one of the hypothesis. 



3. The choice of critical area 

Let the probability of the realization of n events in 
the experiment be described by Poisson distribution 
with parameter A, i.e. 



(6) 



In this case the estimators of Type I and II errors, 
which will take place in testing of hypothesis Hq versus 
hypothesis H±, can be written as follows: 



& = ^2 f(i] n s + n b ) = ^2 /oCO, 
0=1-J2f(i;n b ) = fi(i), 



(7) 



where n c is a critical value. Correspondingly, the mag- 
nitude k = will have minimal value under ap- 
plying of the equal probability test [lj with critical 
value (see, Fig.l) 



(8) 



ln(n s + rib) — ln(rib) ' 



where square brackets mean the integer part of a num- 
ber. It is easy to show that the k has a minimum if 
we require fo(i) — fi(i) (for discrete distributions it 
corresponds to condition fo(i) < fi(i)), i.e. 



value 1 — k under the optimum choice of critical value 
characterises the quality of planned experiment. 

Notice, that such approach works for arbitrary 
distributions (see, Fig. 2), including multidimentional 
ones. 




(9) 



Figure 2: The estimation of uncertainty in hypotheses 
testing for arbitrary distributions. 



4. How to take into account the 
statistical uncertainty in the 
determination of n s and n b ? 

Let the values n s — h s and n b — fib be known 
from Monte Carlo calculations. In this case they are 
random variables. These values can be considered as 
estimators of unknown parameters. Consequently, the 
values n Cl a and [3 are also random variables. It means 
that 1 — k is the estimator of the probability of making 
a correct decision in hypotheses testing. Let us con- 
sider how the uncertainties in the knowledge of n s and 
rib influence the value of magnitude of the Probability 
of Making a Correct Decision in hypotheses testing 
(PMCD) 1 — k. Suppose, as before, that the streams 
of signal and background events are Poisson's. 

Let us write down the density of Gamma distribu- 
tion r 0i „ + i as 2 



It is direct consequence of the equation 



a + p 



1 n c 

= ~(i-£(a« -/<>«))• 



i=0 



(10) 



The value of k decreases with increasing of i from 
up to i — n c . As soon as fo(i) > fi(i) the value of k 
increases. 

Note that the equal probability test gives the results 
close to the results of the equal-tailed test in the case 
of Poisson distributions and we use it hereafter. 

Following the given discourse, we can choose critical 
values so that k could be minimal and the probability 
of correct decision 1 — k - maximum for any pair of 
distributions. As a result it is possible to say, that the 



,71+1. 



A" 
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r(n + i) 

where a is a scale parameter, n + 1 > is a shape 
parameter, A > is a random variable, and r(n + l) = 
n ! is a Gamma function. 

Let us set a = 1 , then for each n a continuous func- 
tion 



9nW 



A" 



A > 0, n > -1 



(12) 



2 Here the traditional designations of Gamma-distribution ■ 
a and x is replaced by a, n + 1 and A, correspondingly. 



is the density of Gamma distribution Fi,„ + i with the 
scale parameter a = 1 (see Fig. 3). The mean, mode, 
and variance of this distribution are given by n + 1, n, 
and Ti + 1, respectively. 

As it follows from the article (see, also 0) and 
is clearly seen from the identity (Fig. 4) 



Figure 4: The Poisson distributions /(n, A) for A's 
determined by the confidence limits Ai = 1.51 and 
A2 = 8.36 in case of the observed number of events n — 4 
are shown. The probability density of Gamma 
distribution with a scale parameter a = 1 and a shape 
parameter n+l = n+ l = 5is shown within this 
confidence interval. 



n— n+1 



00 



Ai 



ln(X)d\+^ /(™; A2) = 1 , i-e. 



n=0 



(13) 



A?e- Al 



n— n+1 



= 1 



71=0 



for any Ai > and A2 > 0, the probability of true 
value of parameter of Poisson distribution to be equal 
to the value of A in the case of one observation h 
has probability density of Gamma distribution ITi i+a. 
The Eq.(13) shows that we can mix Bayesian and fre- 
quentist probabilities in the given approach. This 
identity does not leave a place for any prior except 
uniform. The bounds Ai and A2 hx it. 



Figure 3: The behaviour of the probability density of the 

true value of parameter A for the Poisson distribution in 

case of n observed events versus A and n. Here 
A" 

f(n; A) = g n (A) = — -eT is both the Poisson distribution 

with the parameter A along the axis n and the Gamma 
distribution with a shape parameter n + 1 and a scale 
parameter 1 along the axis A. 





It allows to transform the probability distributions 
f(i;n s + rib) and f(i; rib) accordingly to calculate the 
probability of making a correct decision 



1* OO ,L c (t c 

/ 3„ s+ „ 6 (A)E/( i; A)dA = E 



2n s +n fa +'i+l ' 



4=1-/ .^(A)E/(^)dA = l-^^ r , 

Jo i=a i=0 



1 - k = 1 



a + /3 



(14) 

Here the critical value n c under the future hypothe- 
ses testing about the observability is chosen in accor- 
dance with test of equal probability (Eq.8) and C l N 
N\ 

is — rr. Also we suppose that the Monte Carlo 

i\(N-i)\ 

luminosity is exactly the same as the data luminosity 
later in the experiment. 

The Poisson distributed random values have a prop- 



erty: if d ~ Pois(Xi), i = 1,2,..., m then ~ 

i=l 

rri 

PoisC^Xi). It means that if we have m observations 
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hi, h%, . . ., h m of the same random value £ ~ Pois(X), 
we can consider these observations as one observation 



n. 



of the Poisson distributed random value with 



i=l 

parameter m ■ A. According to Eq.(13) the probability 
of true value of parameter of this Poisson distribu- 
tion has probability density of Gamma distribution 
r i i + ^ m a - Using the scale parameter m one can 

show that the probability of true value of parameter 
of Poisson distribution in the case of m observations 
of the random value £ ~ Pois(X) has probability den- 
sity of Gamma distribution T . . - , i.e. (see 

J m,l+ > . rii ' v 

Eq.(ll)) 



(15) 



Let us assume that the integrated luminosity of 
planned experiment is L and the integrated luminos- 
ity of Monte Carlo data is m ■ L. For instance, we 
can divide the Monte Carlo data into m parts with 
luminosity corresponding to the planned experiment. 
The result of Monte Carlo experiment in this case 
looks as set of m pairs of numbers ( (rib)i, (rib)i + 
(n a )i ), where (rib)i and (n s )i are the numbers of 
background and signal events observed in each part 



of Monte Carlo data. Let us denote Nh — 



{nb)i 



and N s+b = ^ ((n s ) l + (n b )i). Correspondingly (see 
page 98, 1)' ' 



5. A possible way to take into account 
the systematics 

We consider here forthcoming experiments to search 
for new physics. In this case we must take into account 
the systematic uncertainty which have theoretical ori- 
gin without any statistical properties. For example, 
two loop corrections for most reactions at present are 
not known. It means that we can only estimate the 
scale of influence of background uncertainty on the 
observability of signal, i.e. we can point the admissi- 
ble level of uncertainty in theoretical calculations for 
given experiment proposal. 

Suppose uncertainty in the calculation of exact 
background cross section is determined by parame- 
ter 8, i.e. the exact cross section lies in the interval 
((76, CTb(l + 8)) and the exact value of average number 
of background events lies in the interval (rib, rib(l+8)). 
Let us suppose n b 3> n s . In this instance the discovery 
potential is the most sensitive to the systematic un- 
certainties. As we know nothing about possible values 
of average number of background events, we consider 
the worst case Q. Taking into account Eqs.(7) we 
have the formulae 3 



& = f(i;n b + n„) 

i=0 

= !-£/(*; n*(l + *)) 



(17) 



i=0 



1-K = 1 



a + (3 



where n r is 



n = / G(A b+s ,m,A)^/( i; A)dA = 
'o i= o 

\ ^ r i m 

^N B + b +i r 1 y + Ns + b+l , 

i=0 

(3=1- 



G(N b ,m,X)J^f{i;X)d\ 



i=0 



1 - X! G N b +i 



i=0 



(m + iy+frt+i 



As a result, we have a generalized system of equations 
for the case of different luminosity in planned data 
and Monte Carlo data to calculate the PMCD 1 — k = 



1 



a + (3 



The set of values C 



m 



l+N 



N+i 



0,1, 



(m + l) N +>+ 1 ' 



is a negative binomial (Pascal) distribution 
with real parameters N + 1 and -, mean value 



1 + iV 



m 



and variance 



m + 1 
(l + m)(l + A0 



rib 



ln(n s + rib) — ln(n b ■ (1 + 5)) 



(18) 



(16) 6. Conclusions 



In this paper we have considered the probability of 
making a correct decision in hypotheses testing to esti- 
mate the quality of planned experiments. This estima- 
tor allows to measure the distinguishability of models. 
We estimate the influence of statistical uncertainty in 
determination of mean numbers of signal and back- 
ground events and propose a possible way to take into 
account effects of one-sided systematic errors. 



3 Eqs.(17) realize the worst case when the background cross 
section <T(,(1 + <5) is the maximal one, but we think that both 
the signal and the background cross sections are minimal. Also, 
we suppose that rij,(l + <5) < n s + n^. 
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